Utility of RAND/UCLA appropriateness method in validating multiple-choice questions on ECG.

OBJECTIVES: This study aimed to investigate the utility of the RAND/UCLA appropriateness method (RAM) in validating expert consensus-based multiple-choice questions (MCQs) on electrocardiogram (ECG). METHODS: According to the RAM user's manual, nine panelists comprising various experts who routinely handle ECGs were asked to reach a consensus in three phases: a preparatory phase (round 0), an online test phase (round 1), and a face-to-face expert panel meeting (round 2). In round 0, the objectives and future timeline of the study were elucidated to the nine expert panelists with a summary of relevant literature. In round 1, 100 ECG questions prepared by two skilled cardiologists were answered, and the success rate was calculated by dividing the number of correct answers by 9. Furthermore, the questions were stratified into "Appropriate," "Discussion," or "Inappropriate" according to the median score and interquartile range (IQR) of appropriateness rating by nine panelists. In round 2, the validity of the 100 ECG questions was discussed in an expert panel meeting according to the results of round 1 and finally reassessed as "Appropriate," "Candidate," "Revision," and "Defer." RESULTS: In round 1 results, the average success rate of the nine experts was 0.89. Using the median score and IQR, 54 questions were classified as " Discussion." In the expert panel meeting in round 2, 23% of the original 100 questions was ultimately deemed inappropriate, although they had been prepared by two skilled cardiologists. Most of the 46 questions categorized as "Appropriate" using the median score and IQR in round 1 were considered "Appropriate" even after round 2 (44/46, 95.7%). CONCLUSIONS: The use of the median score and IQR allowed for a more objective determination of question validity. The RAM may help select appropriate questions, contributing to the preparation of higher-quality tests.

Protocol for development of an assessment tool for competency of ECG interpretation: expert consensus by the RAND/UCLA appropriateness method and cross-sectional testing using multidimensional item response theory.

INTRODUCTION: Although the ECG is an important diagnostic tool in medical practice, the competency of ECG interpretation is considered to be poor. Diagnostic inaccuracy involving the misinterpretation of ECG can lead to inappropriate medical judgements and cause negative clinical outcomes, unnecessary medical testing and even fatalities. Despite the importance of assessing ECG interpretation skills, there is currently no established universal, standardised assessment tool for ECG interpretation. The current study seeks to (1) develop a set of items (ECG questions) for estimating competency of ECG interpretation by medical personnel by consensus among expert panels following a process based on the RAND/UCLA Appropriateness Method (RAM) and (2) analyse item parameters and multidimensional latent factors of the test set to develop an assessment tool. METHODS AND ANALYSIS: This study will be conducted in two steps: (1) selection of question items for ECG interpretation assessment by expert panels via a consensus process following RAM and (2) cross-sectional, web-based testing using a set of ECG questions. A multidisciplinary panel of experts will evaluate the answers and appropriateness and select 50 questions as the next step. Based on data collected from a predicted sample size of 438 test participants recruited from physicians, nurses, medical and nursing students, and other healthcare professionals, we plan to statistically analyse item parameters and participant performance using multidimensional item response theory. Additionally, we will attempt to detect possible latent factors in the competency of ECG interpretation. A test set of question items for ECG interpretation will be proposed on the basis of the extracted parameters. ETHICS AND DISSEMINATION: The protocol of this study was approved by the Institutional Review Board of Ehime University Graduate School of Medicine (IRB number: 2209008). We will obtain informed consent from all participants. The findings will be submitted for publication in peer-reviewed journals.

Availability of virtual-assisted lung mapping affects procedure selection for early-stage lung cancer: a web-based cross-sectional study.

OBJECTIVES: The availability of new techniques may affect surgeons' procedure selection and thereby affect clinical outcomes. This study aimed to evaluate the effect of the availability of virtual-assisted lung mapping (VAL-MAP) on the selection of lung resection methods. METHODS: Members of the Japanese Association for Chest Surgeons were invited to participate in a web-based survey. Participants were divided into those who had never used VAL-MAP (group 0), those who had used only VAL-MAP 1.0 (multiple dye marks on the pleural surface; group 1) and those who had used VAL-MAP 2.0 (multiple dye marks and intrabronchial microcoils for three-dimensional mapping; group 2). Participants were shown chest computed tomography images of 6 ground-glass opacity nodules and asked to choose surgical procedures to resect the nodules with sufficient resection margins greater than the lesion diameter or 2 cm. RESULTS: There were 197 surgeons in group 0, 49 in group 1 and 26 in group 2. All groups showed a similar trend of avoiding wedge resection for deeply located nodules. However, group 1 showed a trend of disagreeing with the selection of wedge resection compared with group 0 as measured by a Likert scale (1-5) by -0.21 points (95% confidence interval, -0.41 to -0.008 points, P = 0.042). This tendency disappeared in group 2. CONCLUSIONS: The availability of VAL-MAP 1.0 led to the selection of segmentectomy, while the availability of VAL-MAP 2.0 led to aggressive deep wedge resection.

Development and validation of the Japanese version of the uMARS (user version of the mobile app rating system).

BACKGROUND: Although the global market of Mobile Health Apps (mHealth apps) continues to grow dramatically, most mHealth apps still not only lack evidence base but have even not been evaluated for the basic usability or functionality. The User Version of the Mobile App Rating Scale (uMARS) was developed to allow end users to assess mHealth apps objectively and subjectively. However, there is no Japanese version of uMARS to date. OBJECTIVE: The purpose of this study is (1) to develop a validated Japanese version of uMARS and (2) to assess the translated version's reliability and validity in evaluating mHealth apps. METHODS: The original uMARS was adapted for Japanese use by four specialists using universalist cross-cultural methods. Translation/back-translation was reviewed by the author of the original version of uMARS, and confirmed. Its reliability and validity were further evaluated as part of a prospective cohort study of postoperative patients using a new mHealth app. RESULTS: Conceptual equivalence was analyzed and all items in all subcategories of the original uMARS were included in the Japanese version. Internal consistency was deemed acceptable for all subscales of objective and subjective quality with a Cronbach's alpha of 0.75-0.85. Test-retest reliability of all subscales was also acceptable with intraclass correlation coefficients (ICCs) of 0.57-0.88. Convergent/divergent validity and concurrent validity were also considered acceptable. CONCLUSION: A Japanese version of uMARS was cross-culturally validated and found to be as reliable as the original uMARS. This Japanese version of uMARS is expected to become a standard tool in assessing the quality of mHealth apps in Japan.